1. Field of the Invention
The invention relates generally to the field of graphics. More specifically, the invention relates to the field of digital imaging systems and software.
2. Description of the Related Art
In some digital cameras and imaging devices, the color information of a scene is captured by sub-sampling an image typically into three (Red, Green and Blue) color planes. In such cases, the original “raw image” is captured with each “raw image pixel” composed of only one of three vital color/intensity components: either R, G, or B (Red, Green, or Blue). But for image rendering devices (such as a monitor or printer), full color resolution, wherein each pixel is composed of a combination of R, G and B components, components, is desirable. Recovery of full color information from a sub-sampled image of single-color component raw image pixels is popularly known as color interpolation.
This is a continuation of application Ser. No. 09/034,625, filed on Mar. 4, 1998, that is currently pending. The following description refers to raw image pixels as either “pixels” or “raw image pixels”. Pixels that are formed by virtue of a completed color interpolation technique will be referred to as “full color pixels”. In digital still and video cameras and certain other imaging devices, raw images are first represented as rectangular row and column of pixels with each pixel having the intensity value of a particular color plane only. In the case of RGB (Red, Green and Blue) sub-sampling imaging devices, images are obtained and stored in a particular pattern. FIG. 1(a) shows are such pattern, which is considered in describing the invention, known as the “Bayer” pattern. The Bayer pattern (see FIG. 1(a)), when three color planes are sub-sampled, is characterized by including on a first row of an image, alternating pixels of Red and Green, and then on a next row alternating pixels of Blue and Green. Thus, the Bayer pattern for the first four rows of pixels (with each pixel typically an 8-bit value) is as follows (with the rows thereafter repeating the same pattern):
G R G R G R . . . B G B G B G . . . G R G R G R . . . B G B G B G . . .        .        .        .
As a result, each pixel location contains a value for a single color only. Assuming, as is the case in some imaging devices, that each pixel of the Bayer pattern has 8 bits of resolution (i.e., the pixel is a value representing the intensity of the color ranging from 0–255), then a “full color” pixel, one having R, G and B components would be a 24-bit value. To recover the two missing color components for each pixel, various methods of color interpolation or color recovery have been employed.
In color interpolation, an attempt is made to combine in some fashion neighboring pixel(s) to reconstruct, recover, or predict missing color component information for a given raw image pixel. Several prior art interpolation methods are described in “Interactions Between Color Plane Interpolation and Other Image Processing Functions in Electronic Photography”, James E. Adams, Jr., SPIE, (The International Society for Optical Engineering), Vol. 2416 (Electronic Imaging '95), pages 144–155 (July 1995), with most of these employing pixel averaging techniques to recover missing color components. At its simplest, a prior art averaging technique would interpolate the Red component of the captured Green pixel at row 2, column 2, (referring to the 8-bit Bayer pattern above) as the average of the two nearest red pixels, i.e., Red pixel at row 1, column 2 and the Red pixel at row 3, column 2. Likewise, the Blue component of the same captured Green at row 2, column 2 would be interpolated under an averaging technique as the average of the two nearest Blue pixels, i.e., the pixel at row 2, column 1 and the Blue pixel at row 2, column 3. The Blue and Red interpolated components are then combined with the original captured Green value of the pixel, row 2, column 2 to form a new pixel which is a composite “full” color pixel. The resultant “RGB” full color pixel is an approximation or prediction of the original object's true color at that location in the captured scene. FIG. 1(a) illustrates that to store the color interpolated image, three times the memory size of the original Bayer pattern image, would be required, assuming that each color plane component, R, G and B retains the same number of bits after interpolation.
Once each pixel of the image is at a full RGB color resolution (either through color interpolation or by virtue of having been captured at full color resolution), another process known as color space conversion may be performed. Depending on the rendering device or particular application for which the full color image is destined, the RGB “color space” (of an interpolated image, see FIG. 1(a)) may need to be converted. Certain imaging applications such as image editors or rendering devices such as printers do not use RGB color space in manipulating or rendering images. Instead they use different color spaces which may have the advantage of being more standardized or smaller in size. The RGB color space primarily indicates the levels of R, G and B in a pixel location and thus is a measure of the spectrum. An RGB color space image such as one resulting from color interpolating a Bayer pattern image also has both “chrominance” and “luminance” information although this information is less apparent. Luminance refers generally to the level of brightness in an image or pixel, whereas chrominance refers to the hue or placement of color in the color spectrum and saturation or intensity of that hue. In an RGB image, the Green color plane associated pixels contain most of the luminance information while Blue and Red associated pixels contain most of the chrominance information. This is due partly to the response of the human eye to recognize hue variations in the color Green over the colors Red and Blue, which are more associated in the human vision system with brightness or luminance levels. While a fully color interpolated RGB color space is excellent at representing the colors in an image, it is also three times larger in size that the original Bayer pattern (assuming for example an 8-bit to 24-bit color interpolation). Further, in an RGB color space image, the amount of Green is equivalent to the amount Red and Blue, and thus chrominance information may not be adequately represented. For these and other reasons, a color space known as YCrCb was developed.
YCrCb is a digital color system defined by the CCIR (International Consultative Committee on Broadcasting) and is also referred to as the CCIR 601 color space. YCrCb defines color images information in terms of luminance (Y component) and two different chrominance values (Cr—chrominance of Red colors and Cb—chrominance of Blue colors) rather than as a combination of the spectrum like the RGB space. When each pixel is thus represented, unlike the RGB color space where each pixel typically has 24 bits of information (8 bits for each color plane), each pixel can be represented by only 12 bits. This process is more fully described below, but requires an intermediate step where each first pixel has 24-bit YCrCb information and then is decimated to reduce the set to 12-bit values on the average. FIG. 1(b) illustrates the processing step where the 24-bit RGB space is converted into a 24-bit YCrCb space. As seen in FIG. 1(b), the prior art necessitates using a storage array three times the size of the 8-bit Bayer pattern original image. After the 24-bit YCrCb color space is obtained, every other pixel of Cr and Cb components must be decimated row wise and column wise such that only one in four pixels still has all three components. The other three pixels out of four have only the Y component in the 12-bit (so called because the pixels on average have 12-bits of information) YCrCb space. FIG. 1(c) shows this transformation.
Traditionally, the process of color interpolation and color space conversion have been independently performed. First, a raw image is color interpolated. When it is stored or transferred it may also be first compressed or encoded. The process of color space conversion ordinarily occurs in software or in application separate from the imaging system performing the image capture. Often, even color interpolation is performed in software. If performed in hardware, for instance, on a digital camera device, the process of color interpolation would demand a storage array of 24-bits multiplied by the size (number of rows and columns of pixels) of the image.
Color interpolation and color space conversion also both involve a set of computations. In the case of color interpolation, these computations may constitute equal-weight averaging, according to one traditional color interpolation technique, of neighboring raw image pixels. In the case of color space conversion a transformation matrix is utilized to convert each pixel from an RGB color space value to a YCrCb value. This transformation matrix usually consists of nine values determined by a 3×3 conversion matrix. After this transformation matrix is applied, the YCrCb values are decimated to reduce the final image to an average of 12 bits.
It would be desirable to design and provide a technique that integrates the operation of color interpolation and color space conversion into a single operation. Such a technique is advantageous in that a smaller storage array size, one sufficient to hold the 12-bit YCrCb information, rather than the full 24-bit RGB and 24-bit YCrCb information, could be implemented. Further, in a digital camera where color interpolation is performed on-camera, it may be desirable to also perform color space conversion in an integrated manner so that the overall computational complexity and time, as well as image memory size can all be reduced.